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(54) A test development system and method for software with a graphical user interface 



(57) An integrated, visual test design system and 
method for graphical user interlace (GUI) software links 
a test designer, a test design library, and a test genera- 
tion engine with a standard commercial capture/replay 
tool. The system's components provide a human tester 
the capabilities to capture sequences of interactions 
with the software under test (SUT), to visually manipu- 



late and modify the sequences, and to create test de- 
signs that represent multiple individual test sequences. 
These features are supported through an interface that 
gives the tester access to a user model of the SUT, with 
simulations of the windows of the software's GUI, and 
that displays graphical representations of recorded test 
sequences and test designs. 
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Description 



|The present invention rel^ejp^^^e^ ac/ 

grams v f6VtesWg a user interface and, mor e particularly, 
f^sff^^ 

ln"a^bTrTputerized system such as, for example, an 
airline reservation system, the graphical user interface 
is a kind of buffer between a user and the underlying 
system that does the work, such as making airline res- 
ervations, issuing tickets and so forth. Thus, the graph- 
ical interface is what the operator, in this example an 
airline agent or ticket agent, sees on the screen in order 
to operate the airline reservation system. 

Testing the graphical user interface involves (1) 
testing that the functions of the underlying application 
work correctly, such as in the present example, issuing 
correct tickets and sending flights to the right places, 
and (2) testing that the graphical display of all of that 
information is correct so that the user sees things in the 
right places on the screen and that when numbers are 
entered into the interface that the correct number goes 
into the application, that windows are put up in the right 
places and that they appear at the right time and so forth. 

One of the main problems in testing such systems 
results from the typically large number of possible ways 
to operate the system. Typically, there are a large 
number of different paths through the system that rep- 
resent correct use of the system. Basically, testing must 
ensure that all of those paths work correctly; that is, it 
should ensure that the system will respond properly to 
any form of input stimulus. One of the biggest problems 
in testing is constructing enough test cases to be able 
to at least examine the operation of all these different 
possible ways. 

Completely exhaustive testing of all modes is not 
feasible because there are too many possibilities. How- 
ever, thorough testing is essential, yet the sheer magni- 
tude of the job can be overwhelming. It is nevertheless 
desirable to come as close to that as may be practicable. 
Among various known strategies, typically, an attempt 
is made to group together similar kinds of operations 
and have a smaller number of representative tests that 
stand for essentially similar ways of using the system. 

While there exist various strategies to generate and 
to group together similar paths through a system, the 
present invention is useful generally without regard to 
the particular strategy intended to be used but rather is 
more concerned with facilitating the task of carrying out 
such operations. Given the existence of a viable strate- 
gy, there is still a need for an efficient way to execute or 
carry out the strategy. It is desirable to have a way to 
efficiently be able to design test cases and be able to 
readily make variations in a test script, so as to produce 
a slight change in the test script when that slight change 
ma y resu lt in a substantially different test case. 
_Jh£current state of the art metho d fore sting GUis? 
f uses capture/rep jayjools, whjcb„ permit Jhe-testeMo? 



70 



^record,,and:later;playiback^scquences of actions onlh^e^ 
MnteJacj^Too ^ 
^icaUntejffaceja^ 

^S^Lg^ y^ 8 ^ 1 ^^^ using jhejnterface 7 Such tools 
? wiiTtypicaTly capture the uses of the interface, they will 
store them, will store representations of those uses, and 
then allow the opjsratojQoj^eplay those uses a s test c as- 
es. Typically, suchtools also enabTe viewing of the^ap- 

— . - -r: ■♦-S^jr^f^ - • • • / 

t ured rep resentations and the makingof chan gesto the 
capturedTepresentations, so that varMionscan be ere- 



making such 
isjryeryjcur^ 

The chief benefit of these tools is that they relieve 
the tester from the task of explicitly describing the de- 
sired sequences of actions to be performed; the se- 
quences are defined by actually performing them. When 
the actions are carried out, the sequences are recorded 
in a test script language, usually proprietary for each in- 
dividual tool. Moreover, such a proprietary language is 
usually a more or less obscure programming language 
which will be different from case to case. The operator 
requires a working knowledge of such a language and 
of how to manipulate it for implementing changes. Mak- 
ing such changes is akin to writing code, that is, like writ- 
ing programs, so that it is not a facile task. 

Furthermore, with typical existing commercial tools, 
the verification of changes which have been made re- 
quires that the operator run the test and note the results. 

In the current state of the art for testing GUIs, an 
advantage of such record/replay tools whicb.permiUha 
teste rJo,record,-andJater-play-back,-s'equences^of-acr, 
/tio ns-on-the -j nte rface, js4haHhey-re I ieve^^Te^testerrOf 
<jthe task oLexplicitly jescri^ 

of actions to be perfojrnedijhe.sequences^re ciefinecF 7 
^bya ctually perform im TtfteTfvAt the time the actions are 
carried out, the sequences are recorded in the test script 
language. 

However, this advantage raises a concomitant 
drawback: only a single sequence of actions can be re- 
corded at one time. To create 500 test scripts, the tester 
must either record 500 different interactions with the 
system being tested, or else record fewer scripts and 
then create multiple modifications by editing the record- 
ed scripts in the proprietary language. The first alterna- 
tive is extremely labor-intensive and time-consuming, 
while the second requires detailed knowledge of the 
syntax and semantics of the scripting language, and is 
also time-consuming. 

Another problem with the scripts recorded by a 
record/replay tool is keeping them current as the tested 
application changes. While some record/replay tools 
can handle certain simple kinds of changes to the tested 
system, changes in the layout of the interface may in- 
validate many existing test scripts, and require the tester 
either to edit the scripts, or to redevelop a new set of 
scripts from scratch. 

As explained above, the standard available com- 
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mercial technology for testing software with GUI is the 
capture-replay tool. The key functions of these tools are 
to capture information about the structure of a system's 
GUj, to record and store sequences of user interactions 
with the system, and to re-execute the recorded se- 
quences on demand. As noted earlier, they offer either 
no or only rudimentary test generation capabilities. Ex- 
isting capture/replay tools include QA Partner, SQA Ro- 
bot, and XRunnerA/V in Runner. 

A somewhat different approach is utilized in a prior 
art tool the Teradyne TestMaster, which is not a capture/ 
replay tool. TestMaster is a visual tool suite for modeling 
an SUT (software undertest) and generating executable 
test cases. The tester constructs an extended finite state 
machine model of the system being tested. 

The test generator portion of Testmaster creates 
test cases based on paths through the state model. The 
tester can specify the coverage scheme used to gener- 
ate the tests, and can define constraints that limit the 
number and the form of the generated tests. The Test- 
Master system model is an abstraction of the real sys- 
tem's behavior: the generated tests are black-box or 
functional tests. The concepts of coverage used by the 
TestMaster test generator are applied to transitions and 
paths in the system state model, instead of to transitions 
and paths in the code. 

It is her ein reco gnized that the TestMaster system 
and the JOB (Test Develo^ment-Environment)7exhibit 
certain paralIels~sach~asrfor~example7that both utilize 
visual models and both allow the user to place con- 
straints on the test generation process. However, the 
systems differ in other respects. For example, in the 
TestMaster system the system model is in the form of 
an extended finite state machine whereas in TDE is a 
user model; in the TestMaster system, the model is con- 
structed entirely by the user whereas in the TDE, it is 
constructed mainly by the TDE, with some help from the 
user; and the in TestMaster system, tests are generated 
by TestMaster, based on model and coverage criteria 
whereas in TDE the test design is built by the tester and 
actual tests are generated by TDE Test Generation En- 
gine. 

As helpful background material, the following are 
cited: TDE: Test Development Environment for 

Cost-effective Testing of Software with Graphical 
User Interfaces, Siemens Corporate Research, 
Inc., Princeton, U.S.A., November 1996. 
TSL User Guide and Reference Manual Version 
2.2.1, Siemens Corporate Research, 1994. 
QA Partner and QA Planner User's Guide, Segue 
Software, Inc., Newton Centre, MA 1996, 
SQA Robot, http://www.sqa.com/product/suite. 
html, SQA Inc., Burlington, MA. 
Mercury Interactive Home Page, http://www.merc- 
int.com/, Mercury Interactive Corporation, Sunny- 
vale, CA. TestMaster Home Page, 
http ://www. teradyne .com/sst/tm_main . html#test- 
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master, Teradyne Software and Systems Test Prod- 
uct Center, Teradyne Corp. 

/The present invention provides an integrated, visual 
test^esjp/i^ environment fp r^graphicakuser Interface 
(GUI) sbftwarer^The invention links a test designer, a test 
desigh1ibrary,~and a test generation engine with a stand- 
ard commercial capture/replay tool. The invention's 
components provide a human tester the capabilities to 
capture-sequences of interactions with the softvifc^u^ 
d.erJesySLJIJ^^ and modify the se? 

o.uences, and to createjest,designsjiatj:epresent.mul- 
£ipfe inaltfia'uajj^^ ThesTfeatures are~sup£ 

ported through an inteTfacelhat^ives the tester access 
{to a user rnoderof : the SOT, with simulations of the wiiv? 
ciows of the software's GUI, "and that displays graphic^ 
representations of recorded test sequencesand testqie 7 - 

It is an object of the present invention to provide a 
powerful productivity tool that significantly lowers the 
cost of testing software with graphical interfaces. It is 
another object of the present invention to integrate TDE 
with a standard record/replay tool, so as to make it cost- 
effective to develop comprehensive automated tests 
and to update them when the interface changes. 

It is another object of the present invention to enable 
a user to create large quantities of tests for graphical 
user interface (GUI) software with a small fraction of the 
effort formerly required. TDE allows the tester to work 
wi thjest descriptions at a high loy el, to build additiona l 
tests out of parts of existing test s, to construct new tests 
by interacting with the software, and to see how thor- 
oughly a test suite exercises all aspects of the software's 
interface. TDE also simplifies the updating of large 
quantities of test scri pts. 

It is another object of the present inventiojijojam- 
plo v a Record/Rep jayJoolj|i^^tt^^ 
about the_GUI_oH^ 

recor^s^anHnitiaiTet of test scripts"by interacting with 
the software under test. TDE's test designer module 
make s the recorded scripts available for visual editin g . y * 
and^provides access to a library of .possible variations!? ^^^^ 
of qui operations, and input data 



yjs ing JTv^^rintejr 
v fac*e^j^ the. test scripts ana 1 

^their a ssociated datgif After the test design is completed, 
TDE's test generator module processes the test design 
and produces a large number of testscfjp tsinjhej^ 
guage of a record/replay tool. The generated tests can 
be automatically executed to exercise d e siredj vafia- 
so .tions^Awide range of capabilities is available for refining 
test scenarios and for controlling the number and cov- 
erage of the generated test scripts. TDE also gives ac- 
cess to the features of the underlying record/replay tool, 
including validation of test results. 

In accordance with another aspect of the present 
invention an intermediate layer is provided between the 
tester and a standard commercial capture/replay tool; it 
greatly expands the functionality and usefulness of the 
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tool. It provides the tester with a visual representation 
of captured action sequences, a set of easy to use tech- 
niques for manipulating the sequences and creating 
new sequences, and the ability to convert sequences to 
test specifications that can be processed by a test gen- 
eration engine. 

In accordance with an aspect of the invention, an inte- 
grated visual test design method for graphical user in- 
terface ( GUI) software, for lirj Ring aJesT^^igner, a tjjst7 
design library, an d a test g en eration en ginejwitbjajpap- 
^ re/rep lay Jool/comp rises the steps of capturing se- 
quences of user/system interactions with software un- 
der test; visually manipulating the sequences; creating 
test designs thaLrepr.esentjT^pJejndiyjdualJj^t se- 
qu ences; sin^la^ gwindows of the softwa re's GUI^ djs- 
. playing the windows on an interface; an3"d!ispjaying 
J /graphical representations of recorded test sequences 
and test designs/ 

^^IrTa^cordance with another aspect of the invention, 
the step of manipulating the sequences comprises con- 
verting the sequences into Scenario Path Expressions, 
displaying the Scenario Path Expressions for editing, 
and editing. 

In accordance with another aspect of the invention, 
the step of capturing sequences of interactions with soft- 
ware under test comprises the steps of starting the 
graphical user interface (GUI) software; starting the cap- 
ture/replay tool; and sequentially opening and closing 
the windows. 

In accordance with another aspect of the invention, 
an integrated visual test design method for graphical us- 
er interface (GUI) software, for creating a defined test 
development enviro nment for linking a test des igner, a 
|HL^^9IL!!^!iryi-^^est generation engine wrthPa 
capture/replay to^,^c^ ft ^es^!Re^^gs^of captu.rjng 7 
graphical user interfa^jnformatidnTtransforming^ap^ 
^ /tured user interface information into^a^^esentatjoh 
"s u itableTf or the le^ 

a record of user/system interactions; and franstgrming 
the'' rectfrti of uSef/system Int eractions in to Scenario 
Path-Expressions. 

In accordance with another aspect of the invention, 
the step of capturing capturing graphical user interface 
information comprises the steps of starting the graphical 
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user interface (GUI) software; starting the capture/re- 45 

^ ' a ^_^^!i-5? * e ! n ?! ^J??^ 01 e " s ^ ic s °f windbwsoTtfje 

^ft^reTand stotin^the character islicsf 

TheHnvlMtibTrwilf^ understood 
Irom the following detailed description in conjunction 
with the Drawing, in which so 

Figure 1 diagrammatically shows TDE components 
and information flow in accordance with the princi- 
ples of the invention; 

Figure 2 diagrammatically shows the TDE Informa- 55 
tion Capture Phase; 

Figure 3 diagrammatically shows TDE Design/Edit 
Phase in accordance with the principles of the in- 



vention; 

Figure 4 diagrammatically shows steps in con- 
structing TDE display in accordance with the prin- 
ciples of the invention; 

Figure 5 diagrammatically shows steps in SPE (sce- 
nario path expression) processing in accordance 
with the principles of the invention; 
Figure 6 shows a TDE representation of SUTfor an 
embodiment in accordance with the principles of the 
invention; 

Figure 7 shows a TDE Test Manager for an embod- 
iment in accordance with the principles of the inven- 
tion; 

Figure 8 shows a Data Variations window for an em- 
bodiment in accordance with the principles of the 
invention; 

Figure 9 shows a Test Coverage Display on the Text 
Editor for an embodiment in accordance with the 
principles of the invention; and 
Figure 10 shows another example of a TDE Test 
Manager for an embodiment in accordance with the 
principles of the invention. 

The invention will next be described in detail by way 
of an exemplary embodiment, showing the best mode. 
Figure t shows the main components of the TDE sys- 
tem, their interaction with the SUT and a Record/Replay 
tool, and the main steps that occur during use of TDE. 
Initially, the capture/replay tool captures basic informa- 
tion (1) about the GUI of the software under test. The 
tester then records an initial set of test scripts by inter- 
acting with the software (2). TDE's test designer module 
makes the recorded scripts available for visual editing 
(3) and provides access to a library (4) of possible var- 
iations of GUI operations and input data. Using a visual 
interface, the tester specifies how to vary the test scripts 
and their associated data (5). After the test design is 
completed, TDE's test generator module processes the 
test design (6) and produces a large number of test 
scripts (7) in the language of the record/replay took The 
generated tests can be supplied to the capture/replay 
tool, which will execute them on the SUT (8). A wide 
range of capabilities is available for refining test scenar- 
ios and for controlling the number and coverage of the 
generated test scripts. TDE also gives access to the fea- 
tures of the underlying record/replay tool, including val- 
idation of test results. 

TDE represents sequences of user actions with for- 
mulas called Scenario Path Expressions (SPEs). SPEs 
obey a simple grammar whose basic constructs are sin- 
gle actions, action sequences, repeats, and alterna- 
tives. In general, an SPE represents a set of sequences 
ol operations that a user would carry out on the system's 
GUI. A collection of SPEs produced for a GUI is referred 
to as a test design. 

TDE contains a visual editing system lor SPEs, with 
the following features: 
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SPEs are displayed in an entirely visual form to the 
tester, who can cut, paste, and copy in them, as well 
as insert new instances of the basic SPE con- 
structs. 

TDE permits definition of a data variation object for 5 
specifying the possible data values for a GUI oper- 
ation. A data variation object takes on one of several 
values, which themselves may contain other data 
variation objects. An individual data variation object 
may be referenced by several GUI operations, in 10 
several SPEs. 

SPEs can also contain preconditions, which specify 
boundary conditions on the test cases that will be 
generated from the SPE, and constraints, which are 
restrictions on how different parts of the SPE can is 
be combined to form test cases. Both preconditions 
and constraints are used to limit the number of test 
cases that are generated from an SPE. 

j^^gE^aitor is visuaffyihle5^tedw\\h \he appif 20 
^oWbleingll^^Jseractibns on the application can 
be frTSertedlnto any SPE merely by performing the ac- 
tions on a TDE representation of the application. 

The SPE format of a test design is independent of 
both the specific capture/replay tool used and the spe- 25 
cific test generator engine, making test designs com- 
pletely portable across different test development tool- 
sets and execution environments. 

The power of TDE comes from its integration of the 
above capabilities. Prior art methods and tools do not 30 
provide a completely visual system for working with test 
cases and scripts for GUI-based software. 

In accordance with the present invention, from the 
perspective of a user, TDE has three main phases: 1) 
information capture; 2) design/edit of test cases; and 3) 35 
test script generation. 

The information capture phase, phase 1 as referred 
to above, includes TDE's absorbing information about 
the GUI structure that has been captured by the capture/ 
replay tool, and recording sequences of GUI interac- 40 
tions by the tester. 

During the design/edit phase, (phase 2 as referred 
to above), the tester can generalize a single recorded 
interaction sequence to a scenario that represents a set 
of sequences. The possible operations include replac- 45 
ing a single user action with a set of actions, recording 
r andjnsei^n g additio nal user actions into a sequence, 
^ n^sertingjg re -def i n e gj test sequences from the test li- 
brary into*~a sequence, and generalizing the data ele- 
ments that appear in a recorded sequence. so 

During the fesf script generation phase, phase 3 as 
referred to above, the tester specifies which scenarios 
are to be transformed into sets of test scripts and defines 
parameters that control the number of scripts produced 
and the environment in which the scripts are to be run. ss 

In the information capture phase, Figure 2 shows, 
in relerence to an exemplary embodiment of the present 
invention, the steps carried out to record information 



about the SUT's GUI, and to produce the TDE represen- 
tation for the tester. Within each step, the following op- 
erations occur: 

Capture GUI information: the tested application and 
the Record/Replay tool are started. The tester 1 in- 
forms 1 TDE what the windows of the application 
are, by sequentially opening and closing each one. 
As this tour of the SUT's windows goes n, TDE 
builds a script (in the language of the capture/replay 
tool) to visit each window and extract detailed infor- 
mation about the window's structure. 
Transform-eaptured'GUIIhfoTrnation to TDE rep/e- 
sentation: TDE creates the interactions Graph 
showing the re lations . among Jhe.GUI's. windows, 
and a simulation of each^wmdowJo_d isplay_in_th.e 
C omponents Window 

^Record user/system interactions: the standard 
mechanism of the capture/replay tool is used to 
record sequences of user/system interactions. The - 
sequences are stored in the scripting language of 
the capture/replay tool. 

Transform user/system interactions to TDE repre- 
sentations: the recorded sequences are converted 
into SPEs, which are then displayed graphically for 
the tester to edit in the Design/Edit phase. 

In the Design/Edit Phase, see Figure 3, the tester 
works with scenarios and creates a fesf design, which 
can later be converted into executable test scripts by the 
Test Generator Engine. A test design can be built out of 
a recorded sequence of actions that has been converted 
into an SPE, or can be started from scratch. During the 
Design/Edit Phase, the tester can insert variational con- 
structs, such as for example, choose SPE and Data Var- 
iatio n which represent a plurality of a te st cases. 

jrp|^^ con7 ^? 

^ve'rts th etes t de s ign from SPE forrnjn^o^^ pg oTinpjj T? 
specification requirS?byTfhe test g ene ^b7e?^ine7 Th e 
generatoTeTiginelFien parseslhe converted test design 
and produces test scripts that can be executed on the 
a pplicatio n^y^j2e_ca pture/replay tool. T he-p rototype 
fAj|l|^ n jQPE_,uses_the TSL system as ' j^j^erator 
fenginef'See TSL User Guide and Reference Manual 
'Version 3.0, Siemens Corporate Research, 1994. The 
data variation objects, preconditions and constraints of 
an SPE have natural counterparts in the TSL language. 

As regards the TDE internal processing, consider 
first the construction of the TDE displays. TDE uses in- 
formation captured by the capture/replay tool to build 
two structures containing information about the struc- 
ture of the GUI being tested. TDE const ructs.a.GjJLp^?- 
ject fo rest and an Opens Relation. ^ ^Ul^jectjpxe^t 
jgffi^c^ne ^idfc^^ 

Jfce^Gjjj .A top-level windowlsa wlndowlrTaf appears 
on the application's screen not contained in any other 
window. The root node of the tree for a top-level window 
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represents the window itself. Nodes at level 1 represent 
objects directly contained in the window. In general, 
nodes at level i+1 represent objects that are directly con- 
tained in an object at level i. An edge leads from a level 
i node to all level i+1 nodes it contains. 

The Opens relation describes the effect of each us- 
er action in terms of the top-level window(s) in the GUI 
that the action opens. The pair (op, wind) is in the rela- 
tion if performing the op causes win to open. The relation 
is in general many-to-many, as a given window might be 
opened by various different operations, and a given op- 
eration could open several different windows. 

Following creation of the GUI object forest and 
Open, TDE constructs internal representations of the In- 
teractions Graph and the Components Window. To con- 
struct the Interactions graph, each root of a tree in the 
GUI object forest becomes a node. The edges of the 
Interactions Graph are inserted according to the Opens 
relation. The Components Window contains an image 
of the screen display of each window represented in the 
Interactions Graph. These images are constructed from 
the containment information in the GUI object forest. 
The exact location of most widgets in the windows is 
recorded by the capture/ replay tool, and used to build 
the images in the TDE display 

The test scenario representation, creation, and ed- 
iting are next considered. Scenario Path Expressions 
are defined according to the following simple grammar: 

SPE::= action I sequence I alternative I repeat 
SPE-iist : := SPE I SPE-list , SPE action ::= lUser action 
in GUli sequence:- SEQ SPE-list endSEQ 
alternative::- ALT SPE-list endALT repeat::- REP 
pos-integerSPB 

A lUser action in GUI| is any single user step that 
can be performed in the GUI. A pos-integer \s a positive 
integer. The grammar uses endSEQ and endALT to 
prevent ambiguity. In the TDE display, a visual repre- 
sentation that differs from the syntax above is used to 
show SPEs. In particular, the elements of an SPE-list 
are displayed vertically without the comma separators, 
visual indentation is used to show the scope of SEQ, 
ALT, and REP, and the endSEQ and endALT delimiters 
are not used. For examples, see TDE: Test Develop- 
ment Environment for Cost-effective Testing of Software 
with Graphical User Interfaces, Siemens Corporate Re- 
search, Inc., Princeton, U.S.A., November 1 996. 

An SPE represents a set of sequences of user ac- 
tions in the GUI, as follows: 

1. if S = |a| then S represents the set containing 
the single user action iai 

2. if S = SEQ S1 S2 ... Sn endSEQ, then S repre- 
sents the set of sequences that consist of any se- 
quence from S1, followed by any sequence from 
S2, ... followed by any sequence from Sn. 

3. if S = ALT S1 S2 ... Sn endALT, then S repre- 
sents the set ol sequences that is the union of all 
sequences in S1, S2, .... Sn. 



4. if T = REP k S, then T represents the set of se- 
quences of the form s1.s2...sk, where each si is a 
sequence in S. 

5 It is noted that the grammar and its semantics differ 
from the standard grammar for regular expressions only 
in the repetition operator, which is limited to finite repe- 
tition. 

Fig. 5 shows the processing steps of TDE con- 
*o cerned with SPEs. The SPE Converter transforms 
scripts recorded by the capture/replay tool into internal 
SPE representations. Since a recorded script is a se- 
quence of user actions, the resulting SPE is always a 
SEQ, of the form SEQ a b c ... The SPE Editor allows 
is the tester to modify existing SPEs, as well as create new 
ones. The editor uses the information recorded in the 
Opens Relation to provide the connection between the 
objects of the GUI and operations on those objects. 
Since TDE allows different test generator engines, a 
20 test design converter exists as a bridge between the 
TDE SPE format, and each specific test generator en- 
gine. The converter for a given engine transforms SPEs 
into a test specification format that is suitable input for 
that engine. 

25 Among the benefits of TDE, the following are sum- 
marized: 

High-level representation for test sequences. Much 
of TDE's power comes from the SPE representation 
30 of tests. This representation allows visual editing 
and easy manipulation of test libraries, and makes 
it possible to carry out some automated updating of 
test scripts when the system interface is changed. 
The TDE is easy to learn and use. Describing test 
35 scripts and design is based on using the application; 
the test designer does not have to learn a special 
test scripting language. Test structure and varia- 
tions are described in a high-level, easy to under- 
stand graphical format. Test design and specif ica- 
40 tion of test variations are done directly in terms of 
the application, making it quick and relatively inex- 
pensive to develop comprehensive tests. Many 
changes to previously created test scripts can be 
done at a high level, using the TDE visual interface. 
<s Much of the tedious editing work necessary to 
change low-level scripts is avoided, and errors are 
less likely to be made. 

Automated support for test updating: When chang- 
es are made to the design of the interface being 
so tested, the existing test scripts must also be 
changed to remain usable. With existing CR tools, 
many of these changes must be accomplished by 
the labor-intensive process of direct editing of the 
recorded scripts. In TDE, a change to a component 
55 of the system interface can be tracked to the SPEs 
that refer to the changed component. TDE can au- 
tomatically make certain SPE changes, which then 
will propagate to the test scripts generated from 
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those SPE$. 

As for integration with non-GUI testing, the testing 
of the application as well as the GUI are seamlessly 
integrated into test scenarios. Test coverage is 
measured and displayed in terms of user-level op- 
erations in the application. 

Certain terms are used consistently herein with the 
following meanings. 

software under test (SUT): the application soft- 
ware which is being tested. 

user: a user of the software under test. 

tester: an individual who designs, develops, and 
runs tests. A user of testing tools. 

test script: an executable sequence of actions that 
can be supplied as inputs to the SUT. It may contain 
one or more individual test cases. 

(test) scenario: a high-level description of actions 
that could be performed by the SUT A test scenario 
can be converted into a test script by adding what- 
ever details are needed for the SUT to actually ex- 
ecute it. 

scenario group: a generalized test scenario, which 
is an SPE and which may have variations in its con- 
trol flow and its data items. Control flow variations 
allow branching (or alternate paths) in a scenario. 
Data variations allow a data item to take on one of 
a set of possible values. 

In operation, the TDE is described as follows. Fig- 
ures 6,7,8, and 9 show elements of the current TDE pro- 
totype |M re ^^rTaXth e^in5ovfe' contain inglhe^T^E 
re p^eser^ 0 show 

the tools used to develop test scenarios and scenario 
groups. Fig. 9 shows the TDE test coverage display. 
Throughout the example, a simple Text Editor applica- 
tion is used to illustrate the features of TDE. 

The TDE representation the Software under Test 
(SUT) will next be described. Figure 6 shows TDE's 
Control Panel (A), Interactions Window (B), and Com- 
ponent Window (C). The Interactions window contains 
the Interactions graph, in which each node (large oval) 
represents one of the SUT's windows. An arrow from 
one node to another node means that some user oper- 
ation (such as a button click, a menu selection, or a Re- 
turn) perform^djn jhe first window c anopen the_sj3copd 
wjndowTlne^Cp^ used to display 
TDE-bujlUmages oLtbe^a ctual windows jQ ie.SjJTs 
^Gyi. A window's image appears in the Component win- 
dow when the tester clicks on that window's node in the 
lnteractions,graph. - . y 

^Tfte jnterac tjgns.graph^and-the^Component^repre 7 - 



/ se ^ a ^^ s ^^ ^^ ^i^^ information about the GUI 
MhfPis^lle^fe^lo^^du rin g.its JnitiaJ.capturephas e . 
Thisphase requires a tour of the interface, i.e., succes- 
sively invoking each window. In a TDE prototype that 

5 was constructed, this is done manually; in production 
versions the process will be automated. 

Visualizing and Creating Scenarios will next be de- 
scribed. Figure 2 shows TDE's Test Manager (1 ), which 
is used to create new scenarios and edit existing see- 

10 narios. TDE representations of scenario groups can be 
seen in windows (2) and (4). Window (2) contains the 
single scenario that was produced by TDE from one se- 
quence of user actions that was originally captured by 
the record/replay tool. Each diamond-shaped box rep- 

15 resents a single user action. 

Once it has been recorded and converted to a TDE 
scenario, this sequence can become the basis for addi- 
tional scenarios developed by the tester. TDE provides 
a visual correspondence between a scenario and the 

20 representation of the SUT's interface in the Component 
window (3). If the tester clicks on a step in the scenario, 
TDE highlights the corresponding action (button click, 
typing, menu selection, etc.) in the proper place in the 
Component window. 

25 steps for augmenting and modifying scenarios will 
next be described. Scenarios can be added to or gen- 
eralized in TDE. New scenarios can be created and new 
steps can be added to a scenario merely by carrying out 
actions on the TDE representation of the SUT Scenar- 
io ios can be generalized by inserting blocks that indicate 
a sequence, repeat, or choose (SEQ, REP, ALT) of dif- 
ferent actions. The sequence block contains a se- 
quence of user operations. The repeat block is a short- 
hand way to indicate repeated execution of a sequence 

35 of steps in the scenario. The choose block includes a 
number of different possible action sequences that can 
occur at a given point. A choose block represents 
branching of control flow, and a scenario group, or SPE, 
that contains a choose would be transformed into more 

40 than one test script. Window (4) illustrates the symbols 
used for sequence (square), choose (triangle), and re- 
peat (circle). 

Working with a scenario in TDE is extremely simple, and 
does not require any knowledge of a record/replay tool's 
45 test scripting language. See, for example, TSL Test 
Specification Language User Guide, Version 3.0. Sie- 
mens Corp^rate^e^arch ,_Pri ncetp^ 

^Qj^inserted anywhgiejn ^ 

so To modify a scenario, the tester first clicks on an element 
ol the scenario, making it the current insertion point. In 
window (4), the insertion point is the Choose symbol. A 
symbol is inserted by clicking on its button in the tool bar 
at the window's top. An individual operation is inserted 

55 fay simply carrying out the desired operation in the Com- 
ponent window, asjfjhajfi ster were inte r acting with the 
application itself. Again ,_t he tester need sjToJtno wled ge? 
j^ofT he t^t"s^riptl?TQ1anguage. Scenarios can also be ed- 
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ited with typical edit operations like Cut, Copy, and 
Paste. 

Proper selection ol argument values is a majorjac- 
to r in defining e ffective test cases. T DE^JIowslhe tested 
to parameterize any element ol thelsW'i^i_c onte jFil 



user-supplied information 



jicafexamples of suerrin 7 



, 7 ^ , n ? 

formation-are r a"//fe v /Tame"ih a Save File entry widget, -a 
specificchoice outof-a-lisVof radio buttonsrand a ch oice 
selectee) out of the different elements of a listbox widget. 
U ~Mn the-TDE"visual representatiorTof a scenaridTpa- 
rameterizable arguments to a user action are shown in 
smaller type under the text of the action. Clicking on an 
argument brings.up.a Pats Variatio nwindow fSjnFig u re 
7) jn whicMjej^ejtej ca^ 
^gument. ^^ajue^def in ed in,the kQata^riat^ 
canjjfself "be ajjaramete r.^with-additionaLvariations.de- 
Ifine3iof^it7Figure 8 shows the Data Variations window 
being used to define VAR1 in the displayed scenario, 
and a second window being used to define further var- 
iations for VAR2. 

When individual test scripts are produced from a 
scenario, each different value for a given argument will 
be included in a separate script. The exact number of 
scripts produced from a scenario depends on various 
factors controlled by the tester, including the precondi- 
tions and constraints defined for the scenario. One pos- 
sibility is that each different combination of argument 
values in the scenario is used in a separate script; in 
that case, if a scenario has three places with arguments 
defined, containing respectively 2, 3, and 5 argument 

values,t hen 30 scri pt s would be produced. 

The test generatoisection„oLTDE employs a twb-^ 

stage process to convert the scenario groups_of,a-test 
design into executable test scripts for the SjjT. ^FlrstTe? 
scisnario group, or SPE, isJr ansformedJntoj n put to a 
test g enerator tool, and sec ond, theJestgengir ator too L^ 
proo'ul^eTThVfesr^ separation allows the 

tDE"scenario representation to be independent of any 
specific test execution environment and makes scenar- 
ios portable across different operating systems and 
hardware platforms. 

EvalliatlorTonrie thoroughness of test"case*s _ run"on 
an-applicat ibrf rrelpFThTlesteV decio'e^if^flitibnajiests 
^^^^W^andls'Usef urtbl^lp^rtify thTquality of a 
product"! o outside organizations, such as_cu stomers 
and regulatory agencies. Because, different soft ware- 
pjodu15ihirqToups-wiT^ ne edsJn 
presenting c6Vejag e~dga ?TDE provides a set of b^sic 
cove7agen : e^rt 1 that "can be _c ustomiz ed 

bViheltsfeiyThe'fundamental concept is to measure 
1he~exteht to which test scripts exercise all the elements 
ol a system's GUI. 

TDE evaluates two forms of thoroughness for a set 
of test cases, one in terms of the operations that exist 
i n the software's G UI^and one jn.ter ms of the 6e tofdaJa 
variations that the teste rjiasd^fined/ Operation choices 
coverage is the percentage of all possible user actions 
in the GUI that are exercised by the test cases. Data 
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variation coverage is the percentage of all defined data 
values for the actions that occur in the test set that are 
actually used in the test cases. Operation use coverage 
is a broad measure of test adequacy, indicating whether 
any operations of the system have not been tested at 
all. Data variation coverage is a finer measure, indicat- 
ing the extent to which the operations have been exer- 
cised. ^ — — — _ 

/FigurO showsjhe cgyerage^display^on the^Text 
EdjtoJjEac ff^ 

res|3onds-to-one-wtndow oflhe^SUTfandshows theper- 
cenlage^rather^ coverage 
achieved-for-the-windowrThe tester can select a desired 
coverage /eve/for both Operations and Data; these min- 
imum desired coverage percentages are shown as hor- 
izontal lines in the bar graphs, providing a quick view of 
which windows have not been adequately tested. In the 
Interactions graph, each node is colored according to 
whether or not the desired minimum coverage has been 
achieved for the window represented by that node- 
Automated support for test updating will next be de- 
scribed. Keeping large quantities of test cases up-to- 
date with respect to changes in the system being tested 
is a major problem for test developers. A change in the 
SUT's interface might imply that new user operations 
become available, that some previous operations no 
longer exist, or thai an existing operation is carried out 
in a different way or in another submenu. 

ln~somecasesthe-tester-may-wish-to.replace.exist- 
ing_scripts with-new scripts that caty^ ut v 'exact ty^ h e 
sam£:tests:as!previously^ 
ings.hay^b^enim^ 

might mgfn j^iat.an.existingscriptjs.noJonger-meaning - 
I, or that th e set of.existing.se ripts no longeUhoroughly 
^tesjsjh e.systenx7 

If test scripts are produced from a Record/Replay 
tool, the tester must maintain a large volume of scripts 
in an unfamiliar scripting language. Although some 
changes may require only modification of a few entries 
in a table, others may require detailed updates to all the 
relevant scripts. 

TDE stores test information at a higher level in the 
scenario groups, and generates actual executable test 
scripts only after a scenario group has been edited to 
its final form. When the system interface changes, and 
necessitates a change to a scenario group, the change 
will, in general, propagate to many test scripts. Some 
simple changes to scenarios can be made automatically 
by TDE, while others need tester intervention, aided by 
information supplied by TDE. The following describes 
two examples of SUT changes, and howthey are carried 
out in TDE. 

Example 1: A button is renamed from Dismiss to 
Exit. With TDE, the tester loads the affected window into 
the Component display, sets the display to an Edit 
mode, and interactively changes the button name direct- 
ly on the window. TDE then automatically takes care of 
making all needed changes to the scenarios that refer- 
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ence the button, to the Record/Replay tool's database, 
and to the executable test scripts that activate the but- 
ton. 

Example 2: The sequence of user actions needed 
to perform a single logical task is modified. Such a 
change can be carried out easily by TDE if the task is 
encapsulated in a separate block of actions, or in a sep- 
arate isubscenarici that is referenced in a larger sce- 
nario group. Only the affected scenario or block must be 
modified; the changes will carry through to any larger 
scenarios that refer to the affected ones. 

Output Validation will next be described. Test re- 
sults can be compared to expected results and verified 
at any point during execution of a script produced by 
TDE. A validation block can be inserted into test scenar- 
ios, in the same manner as sequence, choose, and re- 
peat blocks. Validation blocks can contain references to 
previously computed and stored results, which could be 
used for comparison against the current state of the soft- 
ware. In general, any validations that can be done by 
the underlying Record/Replay tool can be inserted into 
a scenario. Simple examples of verifications are wheth- 
er a particular menu is displayed, whether a window 
contains a particular text string, and whether a particular 
window is visible. 

E xpected output results can be described jn.s eve ral 
ways. FifstBhe tester can directly^capture the state of 
on^omiore objects in a^wjn^g^w, t agdgtore the state in 
a.validatioTTPlock to u seJor jcom parisojjaa^ 
fes^uftsrSecoT^ insert : v.efification~code 

intoavaltdatbn-block. Such code can be in any availa- 
ble scripting language of the tester's choice, or the 
scripting langu age of the underl yingjgcor d/replay tool. 
Verifi^ioTTcode can be used to p^rjpj rm^oTn^sorg ? 
to carry out plausi4ili_ty.checks.0n the. actuaLresuitsTo? 
to^com pute ex pected-resufts.for^comparison. 

In many cases a correct expected result can be eas- 
ily determined from input data values that are specified 
in a data variation specification; the code in a Nidation 
block can then refer to the data specification to help 
specify the correct response. Similarly, if an expected 
result in a test script depends on which of a set of alter- 
natives of a choose block is taken, the validation code 
can refer to information in the choose block. 

A third way tojggrfo rm validation checking is to us e 
an Exec block, which_ojf6j^_commandJine-interface 
(wrWe~the~rj*sjjlt^^ be 
checkedT^again in a scripting language of the tester's 
choice. For example, the existence of a file can be ver- 
ified, and its contents can be compared to another file. 

Integration with non-GUI testing is a factor to be 
considered. Much of the overall system testing that is 
external to the GUI can also be accomplished through 
TDE. Exec blocks can be used to validate the results of 
operations that do not modify the GUI. For example, the 
effect on a hardware test harness or a system database 
can be captured in an Exec block and checked in a Val- 
idation block. 



The strategy of testing might involve varying the en- 
vironment or other forcing functions to the SUT before 
invoking the GUI. This strategy can be accommodated 
by preceding the GUI operations in the scenario with Ex- 

$ ec blocks which have data variations. For example, the 
Exec block might send instructions to a hardware test 
harness and then invoke the GUI. 

Custom widgets, that is, objects whose definitions 
and methods are not included in the standard vocabu- 

10 lary, can appear in a GUI. Because TDE's generated test 
scripts are interpreted by an underlying Record/Replay 
system, all commands in the scripts must be understood 
by that system. The commercial Record/Replay sys- 
tems have a {standard vocabularyiof widget classes 

is that they recognize and whose operations they can ex- 
ecute. A Record/Replay tool can execute scripts with 
custom widgets after their operations have been defined 
to the tool. After such definitions have been made, TDE 
can be used to define scenarios that refer to custom 

20 widgets, and it wil I gene rate test scripts that can be prop- 
erly interpreted by the Record/Replay tool. 

/i^E/GUhas-a-working prototype~systeTrTlhat illus- 
trates the concept of a yisuah^sjjevelQprne nt sy stem 
for creating tests for graph icafinterf aces, including the 

25 1 nterf aceYTest Manager, and Data Variations, h as be e n 
run on a Sun workstation under Solaris 2.5, using QA 
Partner from Segue Software, Inc. as the underlying 
record/replay tool. See QA Partner and QA Planner Us- 
er's Guide, Segue Software, Inc., Newton Centre, MA 

30 1996. The system can be easily ported to Windows or 
to other platforms, and be integrated with any standard 
commercial record/replay tool. 

The invention has been described by way of various 
embodiments. However it will be apparent to one of skill 

35 in the art that various changes, substitutions of equiva- 
lent functions and parts, and alterations, can be made 
without departing from the spirit of the invention. For ex- 
ample, it is possible to create a higher level operator 
which is implemented by low level descriptions such as 

40 ALT, SEQ, and REP. It will be understood that such 
changes and the like are intended to be within the scope 
of the claims following. 



45 Claims 

1 . An integrated visual test design method for graphi- 
cal user interface (GUI) software, for linking a test 
designer, a test design library, and a test generation 
so engine with a capture/replay tool, comprising the 
steps of: 

capturing sequences of user/system interac- 
tions with software under test; 
55 visually manipulating said sequences; 

creating test designs that represent multiple in- 
dividual test sequences; 
simulating windows of said software's GUI; 



9 

09/14/2004, EAST Version: 1.4.1 



17 



EP 0 869 433 A2 



18 



displaying said windows on an interface; and 
displaying graphical representations of record- 
ed test sequences and test designs. 

2. An integrated visual test design method as recited 
in claim 1, including a step of converting said test 
designs into an input specification suitable for a test 
generator engine. 

3. An integrated visual test design method as recited 
in claim 2, including a step of applying to a test gen- 
erator engine said test designs converted into an 
input specification, for producing executable 
scripts. 

4. An integrated visual test design method as recited 
in claim 2, including a step of applying to a test gen- 
erator engine said test designs converted into an 
input specification for producing scripts executable 
by said capture/replay tool. 

5. An integrated visual test design method as recited 
in claim 2, including a step of applying to a test gen- 
erator engine comprising a TSL (Test Specification 
Language) system said test designs converted into 
an input specification, for producing executable 
scripts. 

6. An integrated visual test design method as recited 
in claim 2, including a step of applying to a test gen- 
erator engine comprising a TSL (Test Specification 
Language) system said test designs converted into 
an input specification, for producing scripts execut- 
able by said capture/replay tool. 

7. An integrated visual test design method as recited 
in claim 1 , wherein said step of visually manipulat- 
ing said sequences comprises converting said se- 
quences into Scenario Path Expressions. 

8. An integrated visual test design method as recited 
in claim 7, wherein said step of visually manipulat- 
ing said sequences comprises displaying said Sce- 
nario Path Expressions for editing. 

9. An integrated visual test design method as recited 
in claim 8, wherein said step of visually manipulat- 
ing said sequences comprises editing. 

10. An integrated visual test design method as recited 
in claim 1 , wherein said step of capturing sequenc- 
es of interactions with software under test compris- 
es the steps of: 

starting said graphical user interface (GUI) soft- 
ware; starting said capture/replay tool; and 
sequentially opening and closing said windows. 



11. An integrated visual test design method as recited 
in claim 10, wherein said step of sequentially open- 
ing and closing said windows comprises the step of 
building a script to visit each of said windows and 

s extract detailed information about said each win- 
dow's structure. 

12. An integrated visual test design method as recited 
in claim 1 1 , wherein said script is in a language uti- 

10 lized by said capture/replay tool. 

13. An integrated visual test design method as recited 
in claim 11, comprising the step of: 

*s transforming said detailed information so as to 

create an Interaction Graph showing relation- 
ships among said windows; and 
creating a simulation ol each of said windows 
for display. 

20 

14. An integrated visual test design method as recited 
in claim 1 , wherein said step of capturing sequenc- 
es of user/system interactions with software under 
test comprises: 

25 

utilizing a standard mechanism of said capture/ 
replay tool to record said sequences of interac- 
tions; and 

storing said sequences. 

30 

15. An integrated visual test design method as recited 
in claim 1 4, wherein said step of storing said se- 
quences comprises storing said sequences in a 
scripting language of said capture/replay tool. 

35 

16. An integrated visual test design method as recited 
in claim 15, comprising the steps of: 

converting said recorded sequences into Sce- 
40 nario Path Expressions; and 

displaying said Scenario Path Expressions. 

17. An integrated visual test design method for graphi- 
cal user interface (GUI) software, for creating a de- 

4 & fined test development environment for linking a 
test designer, a test design library, and a test gen- 
eration engine with a capture/replay tool, compris- 
ing the steps of: 

50 capturing graphical user interface information; 

transforming captured user interface informa- 
tion into a representation suitable for said test 
development environment; 
storing a record of user/system interactions; 
ss and 

transforming said record of user/system inter- 
actions into Scenario Path Expressions. 



20 



25 



30 



SO 



55 



10 

09/14/2004, EAST Version: 1.4.1 



19 



EP 0 869 433 A2 



18. An integrated visual test design method as recited 
in claim 1 7, wherein said step of capturing capturing 
graphical user interface information comprises the 
steps of: 

5 

starting said graphical user interface (GUI) soft- 
ware; starting said capture/replay tool; 
determining characteristics of windows of said 
software; and 

storing said characteristics. 10 



19. An integrated visual test design method as recited 
in claim 1 8, comprising the step of creating a simu- 
lation of each of said windows from said character- 
istics. 15 



20. An integrated visual test design method as recited 
in claim 1 9, comprising the step of creating an In- 
teractions Graph representing relationships be- 
tween said windows and said simulations. 20 

21. An integrated visual test design method as recited 
in claim 17, comprising the steps of: 

recording sequences of user/system interac- 25 
tions by utilizing said capture replay tool; and 
storing said sequences. 

22. An integrated visual test design method as recited 

in claim 21 , wherein said step of storing said se- 30 
quences comprises storing said sequences in a 
scripting language of said capture replay tool. 

23. An integrated visual test design method as recited 

in claim 21 , comprising the steps of: 35 

converting said sequences into Scenario Path 
Expressions; 

displaying said Scenario Path Expressions for 
editing; and 40 
editing. 
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